{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a9bff53e",
   "metadata": {},
   "source": [
    "# 第二章 高级的RAG管道"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "feb98470-c136-471d-a63e-d50d8eb09c57",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .\n",
      "✅ In Context Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Context Relevance, input response will be set to __record__.app.query.rets.source_nodes[:].node.text .\n",
      "✅ In Groundedness, input source will be set to __record__.app.query.rets.source_nodes[:].node.text .\n",
      "✅ In Groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` .\n"
     ]
    }
   ],
   "source": [
    "import utils\n",
    "\n",
    "import os\n",
    "import openai\n",
    "openai.api_key = utils.get_openai_api_key()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ca9cf15",
   "metadata": {},
   "source": [
    "载入文本数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "31e2859b-596e-40b3-867b-f4d6e91f74bc",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index import SimpleDirectoryReader\n",
    "\n",
    "documents = SimpleDirectoryReader(\n",
    "    input_files=[\"data/人工智能.pdf\"]\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2d7d0857-b9d1-4feb-8243-bfd2f4953acd",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> \n",
      "\n",
      "7 \n",
      "\n",
      "<class 'llama_index.schema.Document'>\n",
      "Doc ID: c449baf2-03a3-4f83-a713-f888f7afd7a7\n",
      "Text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称 中国⼤陆⼈⼯智能 台湾⼈⼯智慧\n",
      "港澳⼈⼯智能 新⻢⼈⼯智能、⼈⼯智慧 ⽇韩⼈⼯知能 越南智慧⼈造 [展开] [展开] [展开] [展开] [展开] [展开]⼈⼯智能系列内容\n",
      "主要⽬标 实现⽅式 ⼈⼯智能哲学 历史 技术 术语⼈⼯智能（英语：artiﬁcial intelligence ，缩写为\n",
      "AI）亦称机器智能，指由⼈制造出来的机器所表现出来的智能。通常⼈⼯\n",
      "智能是指⽤普通计算机程序来呈现⼈类智能的技术。该词也指出研究这样的智能系统是否能够实现，以及如何实现。同 时，通过 医学 、神经科学\n",
      "、机器⼈学 及...\n"
     ]
    }
   ],
   "source": [
    "print(type(documents), \"\\n\")\n",
    "print(len(documents), \"\\n\")\n",
    "print(type(documents[0]))\n",
    "print(documents[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "eeaddc7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import SimpleDirectoryReader\n",
    "\n",
    "documents_en = SimpleDirectoryReader(\n",
    "    input_files=[\"data/eBook-How-to-Build-a-Career-in-AI.pdf\"]\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "00880860",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> \n",
      "\n",
      "41 \n",
      "\n",
      "<class 'llama_index.schema.Document'>\n",
      "Doc ID: cd516cc9-b646-4394-ab4e-0465c9090af8\n",
      "Text: PAGE 1Founder, DeepLearning.AICollected Insights from Andrew Ng\n",
      "How to  Build Your Career in AIA Simple Guide\n"
     ]
    }
   ],
   "source": [
    "print(type(documents_en), \"\\n\")\n",
    "print(len(documents_en), \"\\n\")\n",
    "print(type(documents_en[0]))\n",
    "print(documents_en[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3123d3d",
   "metadata": {},
   "source": [
    "## 一、基础RAG通道"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47ec3011",
   "metadata": {},
   "source": [
    "这里通过将 documents 中各个文档的文本连接成一个字符串，然后创建了一个 Document 实例，该实例代表了整个文档集合。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b4abc806-64f5-46bb-8c9f-6469ecb18d20",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index import Document\n",
    "\n",
    "document = Document(text=\"\\n\\n\".join([doc.text for doc in documents]))\n",
    "document_en = Document(text=\"\\n\\n\".join([doc.text for doc in documents_en]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ca7e6dce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将中文标点符号替换成英文标点符号，方便后续处理\n",
    "# 如果是英文文档，可以跳过这一步\n",
    "# 不处理的话，会导致无法正确切分中文句子，会影响后续sentence_window的大小，导致输入长度大于gpt-3.5-turbo的最大限制\n",
    "document.text=document.text.replace('。','. ')\n",
    "document.text=document.text.replace('！','! ')\n",
    "document.text=document.text.replace('？','? ')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "025d24ea",
   "metadata": {},
   "source": [
    "llm-使用 OpenAI 类创建了一个 GPT-3.5-turbo 模型的实例，并设置了温度参数为 0.1。  \n",
    "service_context-使用 ServiceContext 类创建了一个服务上下文实例，包含了前面创建的 GPT-3.5-turbo 模型以及指定的嵌入模型。  \n",
    "index-使用 VectorStoreIndex.from_documents 方法，基于之前创建的文档和服务上下文，创建了一个向量存储索引。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "afc2baff-5e8b-4733-9899-16f248777b23",
   "metadata": {
    "height": 183,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index import VectorStoreIndex\n",
    "from llama_index import ServiceContext\n",
    "from llama_index.llms import OpenAI\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)\n",
    "service_context = ServiceContext.from_defaults(\n",
    "    llm=llm, embed_model=\"local:BAAI/bge-small-zh-v1.5\"\n",
    ")\n",
    "index = VectorStoreIndex.from_documents([document],\n",
    "                                        service_context=service_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "22fc9661",
   "metadata": {},
   "outputs": [],
   "source": [
    "# llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)\n",
    "service_context_en = ServiceContext.from_defaults(\n",
    "    llm=llm, embed_model=\"local:BAAI/bge-small-en-v1.5\"\n",
    ")\n",
    "index_en = VectorStoreIndex.from_documents([document_en],\n",
    "                                        service_context=service_context_en)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea7f3ef7",
   "metadata": {},
   "source": [
    "将之前创建的向量存储索引转换为查询引擎，以便后续进行查询操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ae52a26c-7d0c-44df-8043-4c7f19f794b9",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "query_engine = index.as_query_engine()\n",
    "query_engine_en = index_en.as_query_engine()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df5dd6c4",
   "metadata": {},
   "source": [
    "使用查询引擎执行了一个查询操作，查询给定的问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6b0d5b6e-cc2e-4648-b28c-5fa25a97d175",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "在寻找项目以积累经验时，应该首先明确制定目标并确保能够实现这些目标。然后需要建立一个可预测的世界模型，将整个世界状态用数学模型表现出来，并能够预测它们的行为将如何改变这个世界。在多Agent中，可以通过合作和竞争的方式去完成一定的目标，利用演化算法和群体智能来达成一个整体的突现行为目标。最后，需要通过智能推理来得到新的知识，结合先验知识和特定的推理规则，以便积累更多经验。\n"
     ]
    }
   ],
   "source": [
    "response = query_engine.query(\n",
    "    \"在寻找项目以积累经验时应采取哪些步骤?\"\n",
    ")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "59d8926d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Develop a side hustle, ensure the project will help you grow technically, collaborate with good teammates, and consider if the project can serve as a stepping stone to larger projects.\n"
     ]
    }
   ],
   "source": [
    "response_en = query_engine_en.query(\n",
    "    \"What are steps to take when finding projects to build your experience?\"\n",
    ")\n",
    "print(str(response_en))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e50a1ac5",
   "metadata": {},
   "source": [
    "## 二、使用Trulens进行评测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0ead7dc1-71b2-4001-918f-bf8d610fd3fd",
   "metadata": {
    "height": 132,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "人工智能中的先验知识是如何被存储的？\n",
      "人工智能的自我更新和自我提升是否可能导致其脱离人类的控制？\n",
      "管理者如何管理AI？\n",
      "强人工智能是什么？\n",
      "人工智能被滥用带来的危害？\n"
     ]
    }
   ],
   "source": [
    "eval_questions = []\n",
    "with open('data/eval_questions.txt', 'r') as file:\n",
    "    for line in file:\n",
    "        # Remove newline character and convert to integer\n",
    "        item = line.strip()\n",
    "        print(item)\n",
    "        eval_questions.append(item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c9c2f4ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What are the keys to building a career in AI?\n",
      "How can teamwork contribute to success in AI?\n",
      "What is the importance of networking in AI?\n",
      "What are some good habits to develop for a successful career?\n",
      "How can altruism be beneficial in building a career?\n",
      "What is imposter syndrome and how does it relate to AI?\n",
      "Who are some accomplished individuals who have experienced imposter syndrome?\n",
      "What is the first step to becoming good at AI?\n",
      "What are some common challenges in AI?\n",
      "Is it normal to find parts of AI challenging?\n"
     ]
    }
   ],
   "source": [
    "eval_questions_en = []\n",
    "with open('data/eval_questions_en.txt', 'r') as file:\n",
    "    for line in file:\n",
    "        # Remove newline character and convert to integer\n",
    "        item = line.strip()\n",
    "        print(item)\n",
    "        eval_questions_en.append(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d94b611a",
   "metadata": {},
   "source": [
    "加上自定义问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "87a278f8",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "# You can try your own question:\n",
    "new_question = \"什么是适合我的人工智能工作?\"\n",
    "eval_questions.append(new_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8d5204e8",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['人工智能中的先验知识是如何被存储的？',\n",
       " '人工智能的自我更新和自我提升是否可能导致其脱离人类的控制？',\n",
       " '管理者如何管理AI？',\n",
       " '强人工智能是什么？',\n",
       " '人工智能被滥用带来的危害？',\n",
       " '什么是适合我的人工智能工作?']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "13891936",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['What are the keys to building a career in AI?',\n",
       " 'How can teamwork contribute to success in AI?',\n",
       " 'What is the importance of networking in AI?',\n",
       " 'What are some good habits to develop for a successful career?',\n",
       " 'How can altruism be beneficial in building a career?',\n",
       " 'What is imposter syndrome and how does it relate to AI?',\n",
       " 'Who are some accomplished individuals who have experienced imposter syndrome?',\n",
       " 'What is the first step to becoming good at AI?',\n",
       " 'What are some common challenges in AI?',\n",
       " 'Is it normal to find parts of AI challenging?',\n",
       " 'What is the right AI job for me?']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can try your own question:\n",
    "new_question_en = \"What is the right AI job for me?\"\n",
    "eval_questions_en.append(new_question_en)\n",
    "eval_questions_en"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49afce86",
   "metadata": {},
   "source": [
    "通过调用 reset_database() 方法重置 Trulens 数据库。清空之前的记录和反馈数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c063c9c7-bf1e-4b24-9a22-d4281c0f954e",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🦑 Tru initialized with db url sqlite:///default.sqlite .\n",
      "🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `Tru` to prevent this.\n"
     ]
    }
   ],
   "source": [
    "from trulens_eval import Tru\n",
    "tru = Tru()\n",
    "\n",
    "tru.reset_database()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4363a09b",
   "metadata": {},
   "source": [
    "使用 get_prebuilt_trulens_recorder 函数创建一个 Trulens 记录器 (tru_recorder)，该记录器与给定的查询引擎 (query_engine) 相关联。同时，指定了应用程序的标识为 \"Direct Query Engine\"。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4f754bed-d16f-4c8d-a1a1-b36096272570",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils import get_prebuilt_trulens_recorder\n",
    "\n",
    "tru_recorder = get_prebuilt_trulens_recorder(query_engine,\n",
    "                                             app_id=\"Direct Query Engine\")\n",
    "tru_recorder_en = get_prebuilt_trulens_recorder(query_engine_en,\n",
    "                                             app_id=\"Direct Query Engine_en\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4695f6c",
   "metadata": {},
   "source": [
    "使用 tru_recorder 记录器开始记录过程，遍历 eval_questions 列表，对每个问题进行查询，并将查询引擎的响应记录下来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4dbdfbcc-aac7-4805-9894-4fc016c66bf6",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "with tru_recorder as recording:\n",
    "    for question in eval_questions:\n",
    "        response = query_engine.query(question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b652e1e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "with tru_recorder_en as recording_en:\n",
    "    for question in eval_questions_en:\n",
    "        response_en = query_engine_en.query(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e70334e",
   "metadata": {},
   "source": [
    "获取 Trulens 记录和反馈数据。用于后续分析和评估。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e14f512b-601c-42d0-bfac-bf41d9c577e7",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "records, feedback = tru.get_records_and_feedback(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2da4a602-0d56-4bf8-9fa6-03ef0b7e254b",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app_id</th>\n",
       "      <th>app_json</th>\n",
       "      <th>type</th>\n",
       "      <th>record_id</th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "      <th>tags</th>\n",
       "      <th>record_json</th>\n",
       "      <th>cost_json</th>\n",
       "      <th>perf_json</th>\n",
       "      <th>ts</th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance_calls</th>\n",
       "      <th>Context Relevance_calls</th>\n",
       "      <th>Groundedness_calls</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_tokens</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Direct Query Engine</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_8631870172f29b7facfc339a7c52c465</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u4e2d\\u7684\\u5148\\u9...</td>\n",
       "      <td>\"\\u5148\\u9a8c\\u77e5\\u8bc6\\u5728\\u4eba\\u5de5\\u6...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_8631870172f29b7facf...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T13:22:02.775333\", \"...</td>\n",
       "      <td>2024-03-12T13:22:06.003752</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>[{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...</td>\n",
       "      <td>[{'args': {'source': '[10]\n",
       "早期的⼈⼯智能研究⼈员直接模仿⼈类进⾏...</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Direct Query Engine</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_c7d94d33ddf370a8172a8ce8da3b70ec</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u7684\\u81ea\\u6211\\u6...</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u7684\\u81ea\\u6211\\u6...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_c7d94d33ddf370a8172...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T13:22:06.220520\", \"...</td>\n",
       "      <td>2024-03-12T13:22:08.290158</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.733333</td>\n",
       "      <td>[{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...</td>\n",
       "      <td>[{'args': {'source': '他认为各国应该强制订定规定AI机器只能⽤于⼈类不...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Direct Query Engine</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_80e60e443d0b66da0c5879fde60f2163</td>\n",
       "      <td>\"\\u7ba1\\u7406\\u8005\\u5982\\u4f55\\u7ba1\\u7406AI\\...</td>\n",
       "      <td>\"Management of AI by managers involves treatin...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_80e60e443d0b66da0c5...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T13:22:08.469294\", \"...</td>\n",
       "      <td>2024-03-12T13:22:11.210570</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>[{'args': {'prompt': '管理者如何管理AI？', 'response':...</td>\n",
       "      <td>[{'args': {'prompt': '管理者如何管理AI？', 'response':...</td>\n",
       "      <td>[{'args': {'source': '创造⼒\n",
       "伦理管理\n",
       "经济冲击\n",
       "AI对⼈类的威胁\n",
       "悲...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Direct Query Engine</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_1b323b65311a8cdfa6512765d88c3abb</td>\n",
       "      <td>\"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4ec0\\u4...</td>\n",
       "      <td>\"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4e00\\u7...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_1b323b65311a8cdfa65...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T13:22:11.565023\", \"...</td>\n",
       "      <td>2024-03-12T13:22:14.306188</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>[{'args': {'prompt': '强人工智能是什么？', 'response': ...</td>\n",
       "      <td>[{'args': {'prompt': '强人工智能是什么？', 'response': ...</td>\n",
       "      <td>[{'args': {'source': '⾮⼈类的⼈⼯智能，即机器产⽣了和⼈完全不⼀样的知...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Direct Query Engine</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_54ac38298eed725ca9879c20b65b6ad0</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u88ab\\u6ee5\\u7528\\u5...</td>\n",
       "      <td>\"The misuse of artificial intelligence technol...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_54ac38298eed725ca98...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T13:22:14.764807\", \"...</td>\n",
       "      <td>2024-03-12T13:22:17.776310</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>[{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...</td>\n",
       "      <td>[{'args': {'source': '截⾄2020年12⽉，Sensity检测到的相关...</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                app_id                                           app_json  \\\n",
       "0  Direct Query Engine  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "1  Direct Query Engine  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "2  Direct Query Engine  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "3  Direct Query Engine  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "4  Direct Query Engine  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "\n",
       "                                                type  \\\n",
       "0  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "1  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "2  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "3  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "4  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "\n",
       "                                      record_id  \\\n",
       "0  record_hash_8631870172f29b7facfc339a7c52c465   \n",
       "1  record_hash_c7d94d33ddf370a8172a8ce8da3b70ec   \n",
       "2  record_hash_80e60e443d0b66da0c5879fde60f2163   \n",
       "3  record_hash_1b323b65311a8cdfa6512765d88c3abb   \n",
       "4  record_hash_54ac38298eed725ca9879c20b65b6ad0   \n",
       "\n",
       "                                               input  \\\n",
       "0  \"\\u4eba\\u5de5\\u667a\\u80fd\\u4e2d\\u7684\\u5148\\u9...   \n",
       "1  \"\\u4eba\\u5de5\\u667a\\u80fd\\u7684\\u81ea\\u6211\\u6...   \n",
       "2  \"\\u7ba1\\u7406\\u8005\\u5982\\u4f55\\u7ba1\\u7406AI\\...   \n",
       "3  \"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4ec0\\u4...   \n",
       "4  \"\\u4eba\\u5de5\\u667a\\u80fd\\u88ab\\u6ee5\\u7528\\u5...   \n",
       "\n",
       "                                              output tags  \\\n",
       "0  \"\\u5148\\u9a8c\\u77e5\\u8bc6\\u5728\\u4eba\\u5de5\\u6...    -   \n",
       "1  \"\\u4eba\\u5de5\\u667a\\u80fd\\u7684\\u81ea\\u6211\\u6...    -   \n",
       "2  \"Management of AI by managers involves treatin...    -   \n",
       "3  \"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4e00\\u7...    -   \n",
       "4  \"The misuse of artificial intelligence technol...    -   \n",
       "\n",
       "                                         record_json  \\\n",
       "0  {\"record_id\": \"record_hash_8631870172f29b7facf...   \n",
       "1  {\"record_id\": \"record_hash_c7d94d33ddf370a8172...   \n",
       "2  {\"record_id\": \"record_hash_80e60e443d0b66da0c5...   \n",
       "3  {\"record_id\": \"record_hash_1b323b65311a8cdfa65...   \n",
       "4  {\"record_id\": \"record_hash_54ac38298eed725ca98...   \n",
       "\n",
       "                                           cost_json  \\\n",
       "0  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "1  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "2  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "3  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "4  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "\n",
       "                                           perf_json  \\\n",
       "0  {\"start_time\": \"2024-03-12T13:22:02.775333\", \"...   \n",
       "1  {\"start_time\": \"2024-03-12T13:22:06.220520\", \"...   \n",
       "2  {\"start_time\": \"2024-03-12T13:22:08.469294\", \"...   \n",
       "3  {\"start_time\": \"2024-03-12T13:22:11.565023\", \"...   \n",
       "4  {\"start_time\": \"2024-03-12T13:22:14.764807\", \"...   \n",
       "\n",
       "                           ts  Answer Relevance  Context Relevance  \\\n",
       "0  2024-03-12T13:22:06.003752               1.0                0.6   \n",
       "1  2024-03-12T13:22:08.290158               1.0                0.8   \n",
       "2  2024-03-12T13:22:11.210570               0.8                0.6   \n",
       "3  2024-03-12T13:22:14.306188               1.0                0.9   \n",
       "4  2024-03-12T13:22:17.776310               0.9                0.9   \n",
       "\n",
       "   Groundedness                             Answer Relevance_calls  \\\n",
       "0      1.000000  [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...   \n",
       "1      0.733333  [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...   \n",
       "2      0.000000  [{'args': {'prompt': '管理者如何管理AI？', 'response':...   \n",
       "3      0.500000  [{'args': {'prompt': '强人工智能是什么？', 'response': ...   \n",
       "4      0.666667  [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...   \n",
       "\n",
       "                             Context Relevance_calls  \\\n",
       "0  [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...   \n",
       "1  [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...   \n",
       "2  [{'args': {'prompt': '管理者如何管理AI？', 'response':...   \n",
       "3  [{'args': {'prompt': '强人工智能是什么？', 'response': ...   \n",
       "4  [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...   \n",
       "\n",
       "                                  Groundedness_calls  latency  total_tokens  \\\n",
       "0  [{'args': {'source': '[10]\n",
       "早期的⼈⼯智能研究⼈员直接模仿⼈类进⾏...        3             0   \n",
       "1  [{'args': {'source': '他认为各国应该强制订定规定AI机器只能⽤于⼈类不...        2             0   \n",
       "2  [{'args': {'source': '创造⼒\n",
       "伦理管理\n",
       "经济冲击\n",
       "AI对⼈类的威胁\n",
       "悲...        2             0   \n",
       "3  [{'args': {'source': '⾮⼈类的⼈⼯智能，即机器产⽣了和⼈完全不⼀样的知...        2             0   \n",
       "4  [{'args': {'source': '截⾄2020年12⽉，Sensity检测到的相关...        3             0   \n",
       "\n",
       "   total_cost  \n",
       "0         0.0  \n",
       "1         0.0  \n",
       "2         0.0  \n",
       "3         0.0  \n",
       "4         0.0  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "records.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "627783dc",
   "metadata": {},
   "source": [
    "运行 Trulens 仪表板以可视化评估结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64310897-179b-4081-aab8-f08a3392a078",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# launches on http://localhost:8501/\n",
    "tru.run_dashboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4eedcef",
   "metadata": {},
   "source": [
    "## 三、高级的RAG通道"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a17ea2b",
   "metadata": {},
   "source": [
    "### 3.1 滑窗句子检索"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "665314f2",
   "metadata": {},
   "source": [
    "创建 OpenAI 的 GPT-3.5-turbo 语言模型实例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "dae4a668-3699-4750-82f7-e53ae1bca3a7",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.llms import OpenAI\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7838c82",
   "metadata": {},
   "source": [
    "使用辅助函数 build_sentence_window_index 创建基于窗口的句子索引："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "78f7678f-358d-448d-b153-11ac8e96a7fc",
   "metadata": {
    "height": 149,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils import build_sentence_window_index\n",
    "\n",
    "sentence_index = build_sentence_window_index(\n",
    "    document,\n",
    "    llm,\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",\n",
    "    save_dir=\"sentence_index\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "29193915",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_index_en = build_sentence_window_index(\n",
    "    document_en,\n",
    "    llm,\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
    "    save_dir=\"sentence_index_en\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f685d30c",
   "metadata": {},
   "source": [
    "使用辅助函数 get_sentence_window_query_engine 获取基于句子窗口的查询引擎："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "72f904c3-9845-4df5-9d2e-e5115160f987",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils import get_sentence_window_query_engine\n",
    "\n",
    "sentence_window_engine = get_sentence_window_query_engine(sentence_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "fbcdba73",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_window_engine_en = get_sentence_window_query_engine(sentence_index_en)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37015871",
   "metadata": {},
   "source": [
    "对一个特定的问题进行查询并打印结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f59e2314-7cac-42f4-a552-9a8e4db641eb",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'通过模仿人类思考模式，尝试逐步推理是一个开始人工智能个人项目的方法。另外，利用概率和经济学概念处理不确定或不完整的信息也是一个成功的方法。在解决困难问题时，寻找更有效的算法是优先考虑的。最终，强调感知运动的重要性也是一个可以考虑的方向。'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "window_response = sentence_window_engine.query(\n",
    "    \"如何开始人工智能个人项目?\"\n",
    ")\n",
    "str(window_response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "55235a2a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"To get started on a personal project in AI, it is important to first identify a project that aligns with your career goals and interests. Once you have chosen a project, you can begin by scoping it out, defining the objectives, and outlining the steps needed to achieve them. It's essential to ensure that the project is responsible, ethical, and beneficial to people. As you work on the project, aim to grow in terms of scope, complexity, and impact over time. Building a portfolio of projects that demonstrate skill progression can also be valuable for your career development in AI.\""
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "window_response_en = sentence_window_engine_en.query(\n",
    "    \"how do I get started on a personal project in AI?\"\n",
    ")\n",
    "str(window_response_en)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb980a71",
   "metadata": {},
   "source": [
    "重置 Trulens 数据库，  \n",
    "使用 Trulens 记录器对基于窗口的句子索引进行评估，记录查询结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b5c10917-8846-4e73-838d-6232c906a7db",
   "metadata": {
    "height": 115,
    "tags": []
   },
   "outputs": [],
   "source": [
    "tru.reset_database()\n",
    "\n",
    "tru_recorder_sentence_window = get_prebuilt_trulens_recorder(\n",
    "    sentence_window_engine,\n",
    "    app_id = \"Sentence Window Query Engine\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f472b330",
   "metadata": {},
   "outputs": [],
   "source": [
    "tru_recorder_sentence_window_en = get_prebuilt_trulens_recorder(\n",
    "    sentence_window_engine_en,\n",
    "    app_id = \"Sentence Window Query Engine_en\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "11710e67-aba8-479e-8585-c4c611e2c1d2",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "人工智能中的先验知识是如何被存储的？\n",
      "人工智能中的先验知识是通过某种方式告知机器的知识，可以描述目标、特征、种类及对象之间的关系，也可以描述事件、时间、状态、原因和结果，以及任何需要机器存储的知识。\n",
      "人工智能的自我更新和自我提升是否可能导致其脱离人类的控制？\n",
      "人工智能的自我更新和自我提升可能导致其脱离人类的控制。\n",
      "管理者如何管理AI？\n",
      "Managers can manage AI by considering the following suggestions:\n",
      "1. Delegate administrative tasks.\n",
      "2. Focus on enhancing their comprehensive judgment and creativity in the field of analysis and prediction.\n",
      "3. Treat AI as a colleague and form a collaborative team.\n",
      "4. Recognize that all technologies, including AI, have limitations and may face bottlenecks.\n",
      "强人工智能是什么？\n",
      "强人工智能是一种观点，认为计算机本身具有思维，而不仅仅是用来模拟人类思维的工具。根据这个观点，只要计算机运行适当的程序，它们就具有自己的思维能力。\n",
      "人工智能被滥用带来的危害？\n",
      "The misuse of artificial intelligence could potentially lead to violations of copyright laws and other legal regulations. There have been cases where artificial intelligence technology has been used to remove mosaic from explicit videos, alter the appearance of individuals in videos, and other related incidents. Additionally, experts have raised concerns about the potential misuse of artificial intelligence in manipulating financial markets, scientists, human leaders, and even developing weapons that may be beyond human comprehension and control. Furthermore, there are predictions that certain professions could be replaced by machines and artificial intelligence in the future, impacting millions of individuals and potentially leading to social instability.\n",
      "什么是适合我的人工智能工作?\n",
      "适合您的人工智能工作是那些需要您具备观察周围环境、作出行动以达成目标的能力，同时能够正确解释外部数据、学习并利用知识实现特定目标和任务的工作。这样的工作可能涉及模仿人类思维相关的认知功能，包括学习和解决问题，并需要您具备感知环境、采取行动以最大程度提高成功机会的能力。\n"
     ]
    }
   ],
   "source": [
    "for question in eval_questions:\n",
    "    with tru_recorder_sentence_window as recording:\n",
    "        response = sentence_window_engine.query(question)\n",
    "        print(question)\n",
    "        print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "70471e64",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function BaseQueryEngine.query at 0x000001E7720134C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.indices.vector_store.retrievers.retriever.VectorIndexRetriever'> at 0x1e716c1fa90 is calling an instrumented method <function BaseRetriever.retrieve at 0x000001E77483AD40>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app.retriever based on other object (0x1e71699c5d0) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function CompactAndRefine.get_response at 0x000001E77480B1A0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What are the keys to building a career in AI?\n",
      "Understanding the characteristics of intelligent systems, studying introductory materials on artificial intelligence, and gaining expertise in problem solving, puzzle solving, game playing, and deduction are key elements to building a career in AI.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How can teamwork contribute to success in AI?\n",
      "Teamwork can contribute to success in AI by treating AI as a colleague and forming a collaborative team. This approach fosters synergy and cooperation, allowing for a more effective utilization of AI technologies within various processes and creative endeavors.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is the importance of networking in AI?\n",
      "Networking in AI is crucial as it allows for the exchange of information and collaboration among researchers, experts, and professionals in the field. Through networking, individuals can share insights, best practices, and advancements in artificial intelligence, fostering innovation and progress in the development of AI technologies. Additionally, networking provides opportunities for partnerships, joint projects, and knowledge sharing, which are essential for the growth and evolution of AI applications across various industries and domains.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What are some good habits to develop for a successful career?\n",
      "Developing habits such as relinquishing administrative tasks and focusing on enhancing comprehensive judgment skills can be beneficial for a successful career.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How can altruism be beneficial in building a career?\n",
      "Altruism can be beneficial in building a career by fostering positive relationships, creating a supportive network, and enhancing one's reputation within a professional community. It can also lead to opportunities for collaboration, mentorship, and personal growth, ultimately contributing to long-term career success.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is imposter syndrome and how does it relate to AI?\n",
      "Imposter syndrome is a psychological pattern where individuals doubt their accomplishments and have a persistent fear of being exposed as a fraud. In the context of AI, imposter syndrome can manifest among researchers or professionals who may feel inadequate or fraudulent in their work within the highly technical and specialized field of artificial intelligence. This feeling may arise due to the complexity and vast scope of AI research, leading individuals to question their own abilities and knowledge in comparison to the breadth of the field.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Who are some accomplished individuals who have experienced imposter syndrome?\n",
      "Stephen Hawking and Elon Musk are some accomplished individuals who have experienced imposter syndrome.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is the first step to becoming good at AI?\n",
      "Studying and understanding the complex mathematical tools developed in artificial intelligence research to solve specific branch problems is the first step to becoming proficient in AI.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What are some common challenges in AI?\n",
      "Some common challenges in AI include the fragmentation of AI into subfields that may not communicate effectively with each other, difficulties in areas such as vision, natural language processing, decision theory, genetic algorithms, and robotics, and the potential ethical concerns surrounding the development and deployment of AI technologies.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is it normal to find parts of AI challenging?\n",
      "It is common for individuals to find certain aspects of Artificial Intelligence challenging.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new object of type <class 'llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine'> at 0x1e71800dbd0 is calling an instrumented method <function RetrieverQueryEngine.retrieve at 0x000001E779B307C0>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app based on other object (0x1e7167b0190) using this function.\n",
      "A new object of type <class 'llama_index.response_synthesizers.compact_and_refine.CompactAndRefine'> at 0x1e717f6fd90 is calling an instrumented method <function Refine.get_response at 0x000001E774838860>. The path of this call may be incorrect.\n",
      "Guessing path of new object is app._response_synthesizer based on other object (0x1e7167b01d0) using this function.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is the right AI job for me?\n",
      "A suitable AI job for you would involve working in the field of data science or artificial intelligence. These areas are in high demand and offer various opportunities for individuals with the right skills and expertise. Consider roles such as data scientist, AI specialist, machine learning engineer, or AI researcher, depending on your interests and qualifications.\n"
     ]
    }
   ],
   "source": [
    "for question in eval_questions_en:\n",
    "    with tru_recorder_sentence_window_en as recording:\n",
    "        response = sentence_window_engine.query(question)\n",
    "        print(question)\n",
    "        print(str(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d91b12ed",
   "metadata": {},
   "source": [
    "获取性能评估的排行榜："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "869d1e55-729b-45f2-a0f9-773c49d4616f",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>app_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Sentence Window Query Engine</th>\n",
       "      <td>0.811111</td>\n",
       "      <td>0.916667</td>\n",
       "      <td>0.741667</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sentence Window Query Engine_en</th>\n",
       "      <td>0.300000</td>\n",
       "      <td>0.920000</td>\n",
       "      <td>0.070000</td>\n",
       "      <td>3.727273</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Groundedness  Answer Relevance  \\\n",
       "app_id                                                            \n",
       "Sentence Window Query Engine         0.811111          0.916667   \n",
       "Sentence Window Query Engine_en      0.300000          0.920000   \n",
       "\n",
       "                                 Context Relevance   latency  total_cost  \n",
       "app_id                                                                    \n",
       "Sentence Window Query Engine              0.741667  4.000000         0.0  \n",
       "Sentence Window Query Engine_en           0.070000  3.727273         0.0  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tru.get_leaderboard(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b92d0f2-2e80-48d5-92af-b3655eb03ea2",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# launches on http://localhost:8501/\n",
    "tru.run_dashboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a92e2c55",
   "metadata": {},
   "source": [
    "### 3.2 自动合并检索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "558c639b-31eb-4c34-b6c4-fe6ae5717733",
   "metadata": {
    "height": 149,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils import build_automerging_index\n",
    "\n",
    "automerging_index = build_automerging_index(\n",
    "    documents,\n",
    "    llm,\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",\n",
    "    save_dir=\"merging_index\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "d173c398",
   "metadata": {},
   "outputs": [],
   "source": [
    "automerging_index_en = build_automerging_index(\n",
    "    documents_en,\n",
    "    llm,\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
    "    save_dir=\"merging_index_en\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b32265f2-0247-42df-9abe-97d52f69edcf",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils import get_automerging_query_engine\n",
    "\n",
    "automerging_query_engine = get_automerging_query_engine(\n",
    "    automerging_index,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "ff78b83a",
   "metadata": {},
   "outputs": [],
   "source": [
    "automerging_query_engine_en = get_automerging_query_engine(\n",
    "    automerging_index_en,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "918ed568-220e-4c7c-aa60-cfa58ef1fcbd",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Begin a personal artificial intelligence project by first identifying a specific problem or application you want to work on. Then, explore various tools and technologies that can help you achieve your project goals. Design and create your project with a focus on utilizing different tools to implement the desired application effectively.\n"
     ]
    }
   ],
   "source": [
    "auto_merging_response = automerging_query_engine.query(\n",
    "    \"如何开始人工智能个人项目?\"\n",
    ")\n",
    "\n",
    "print(str(auto_merging_response))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "5437a9ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To get started on a personal project in AI, communicating the value of what you hope to build can help bring colleagues, mentors, and managers onboard. This will also help them point out any flaws in your reasoning. After finishing the project, being able to clearly explain what you accomplished will help convince others to open the door to larger projects. Additionally, starting with small projects in your spare time and achieving initial successes, no matter how small, can help build your skills and increase your ability to come up with better ideas. Joining existing projects or finding someone with an idea to collaborate with can also be a good way to kickstart your personal project in AI.\n"
     ]
    }
   ],
   "source": [
    "auto_merging_response_en = automerging_query_engine_en.query(\n",
    "    \"how do I get started on a personal project in AI?\"\n",
    ")\n",
    "print(str(auto_merging_response_en))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "db4f18a9-7b8a-4ae2-ab11-3a6a941a5afc",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [],
   "source": [
    "tru.reset_database()\n",
    "\n",
    "tru_recorder_automerging = get_prebuilt_trulens_recorder(automerging_query_engine,\n",
    "                                                         app_id=\"Automerging Query Engine\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "04a805bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "tru_recorder_automerging_en = get_prebuilt_trulens_recorder(automerging_query_engine_en,\n",
    "                                                         app_id=\"Automerging Query Engine_en\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "99cc2cfe-7096-4fa0-aa72-094bebac35a3",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Merging 4 nodes into parent node.\n",
      "> Parent node id: 610d624d-381b-4f98-a6df-b6d88b59cca8.\n",
      "> Parent node text: [16]\n",
      "⼈类解决问题的模式通常是⽤最快捷、直观的判断，⽽不是有意识的、⼀步⼀步的推导，早期⼈⼯智能研究通常使⽤逐步推导的⽅式。[17]⼈⼯智能研究已经\n",
      "于这种“次表征性的”解决问题⽅法获取进展...\n",
      "\n",
      "人工智能中的先验知识是如何被存储的？\n",
      "In artificial intelligence, prior knowledge is stored in a way that allows machines to retain descriptions of objectives, features, categories, and relationships between objects. It can also encompass descriptions of events, time, states, causes, results, or any knowledge that one wishes the machine to store. This stored knowledge can include both explicitly provided information and knowledge derived through intelligent reasoning processes.\n",
      "人工智能的自我更新和自我提升是否可能导致其脱离人类的控制？\n",
      "Yes, the self-updating and self-improving capabilities of artificial intelligence could potentially lead to it surpassing human control.\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 0e4eeea0-999b-4905-9fce-9f7fa045325d.\n",
      "> Parent node text: 依⽬前的研究⽅向，电脑⽆法突变、苏醒、产⽣⾃我意志，AI也不可能具有创意与智能、同情⼼与审美等这⽅⾯的能⼒。\n",
      "AI逐渐普及后，将会在企业管理中扮演很重要的⾓⾊，⽽⼈类的管理者应 如何适度的调整⾃...\n",
      "\n",
      "管理者如何管理AI？\n",
      "Management should consider adjusting their work functions by relinquishing administrative tasks, focusing on enhancing their comprehensive judgment abilities, and emphasizing creativity, ethical management, and economic impacts. It is important for managers to understand the potential threats of AI and be prepared to adapt to the changing landscape of AI integration in business management.\n",
      "强人工智能是什么？\n",
      "强人工智能是指一种观点，认为可以制造出真正能够推理和解决问题的智能机器，这样的机器被认为具有知觉和自我意识。\n",
      "人工智能被滥用带来的危害？\n",
      "The misuse of artificial intelligence technology can potentially violate copyright laws and other legal regulations. There have been instances where law enforcement has intervened due to the use of AI technology to remove mosaic from explicit images or alter the content of images. Additionally, the misuse of AI can lead to societal instability if the number of individuals negatively impacted is significant, potentially creating a destabilizing force that could be exploited by malicious actors, resulting in various incidents.\n",
      "什么是适合我的人工智能工作?\n",
      "适合您的人工智能工作是那些需要感知环境、采取行动、从经验中学习、做出合理决策并快速回应的工作，以及可能涉及制造真正能推理和解决问题的智能机器的工作。\n"
     ]
    }
   ],
   "source": [
    "for question in eval_questions:\n",
    "    with tru_recorder_automerging as recording:\n",
    "        response = automerging_query_engine.query(question)\n",
    "        print(question)\n",
    "        print(response)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "2888b4a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 4daa536d-f162-4f96-9724-825d766babe2.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9a1060af-8768-4591-9737-e6295af6ad39.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "What are the keys to building a career in AI?\n",
      "The keys to building a career in AI are learning foundational technical skills, working on projects to deepen skills and create impact, and finding a job in the field. Being part of a community also supports these steps.\n",
      "How can teamwork contribute to success in AI?\n",
      "Teamwork can contribute to success in AI by enabling individuals to collaborate effectively, influence others, and be influenced by team members. Working in teams on large projects in AI can enhance the overall success as it allows for diverse perspectives, shared knowledge, and collective problem-solving capabilities.\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 2a3c89c2-9e1f-4aa1-a041-a8e3f7b545de.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: af74273b-3def-4687-9aec-2ed8e82819b9.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "What is the importance of networking in AI?\n",
      "Networking in AI is crucial as it helps individuals build a strong professional network within the industry. This network can provide support, guidance, and opportunities for career advancement. By connecting with others in the field, individuals can gain valuable insights, stay updated on industry trends, and potentially find mentors who can help them navigate their career paths effectively.\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 6235c1a4-7061-4e82-a58e-8090b4c9aa1f.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: bef5e149-069f-4a76-99c0-a35bcdc8d2f7.\n",
      "> Parent node text: PAGE 11\n",
      "The Best Way to Build \n",
      "a New Habit\n",
      "One of my favorite books is BJ Fogg’s, Tiny Habits: Th...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 40b951a1-f670-423d-90d8-a4133bea59f9.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 236499b9-afbc-4048-bf4c-f4abe1d1f6ed.\n",
      "> Parent node text: PAGE 11\n",
      "The Best Way to Build \n",
      "a New Habit\n",
      "One of my favorite books is BJ Fogg’s, Tiny Habits: Th...\n",
      "\n",
      "What are some good habits to develop for a successful career?\n",
      "Good habits to develop for a successful career include habits related to eating well, exercising, getting enough sleep, maintaining personal relationships, staying dedicated to work, continuous learning, and self-care. These habits can help individuals progress in their careers while also ensuring their overall well-being.\n",
      "How can altruism be beneficial in building a career?\n",
      "Altruism can be beneficial in building a career by helping individuals lift others during their journey, which often leads to achieving better outcomes for themselves. This approach of supporting and helping others can create a positive network and community that can provide assistance and guidance when needed, ultimately propelling one forward in their career.\n",
      "> Merging 5 nodes into parent node.\n",
      "> Parent node id: 078fd2d1-a75f-46ea-80a2-a304afd202a1.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: cafdfa80-e3ec-4877-825a-974ab062bd7c.\n",
      "> Parent node text: PAGE 37Overcoming Imposter \n",
      "SyndromeCHAPTER 11\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: cb9e385b-2d74-46b9-8452-3d41cd2609ef.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 33a81b18-db5a-4170-b2dd-9746b79325d7.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 977dfbae-8938-4e93-a6cc-ea55a9a46cce.\n",
      "> Parent node text: PAGE 37Overcoming Imposter \n",
      "SyndromeCHAPTER 11\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 07b83924-cf67-4952-bf27-b07f1b10e439.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "What is imposter syndrome and how does it relate to AI?\n",
      "Imposter syndrome is a phenomenon where individuals doubt their accomplishments and have a persistent fear of being exposed as a fraud, despite evidence of their competence. In the context of AI, newcomers to the field may experience imposter syndrome due to the technical complexity and the presence of highly capable individuals within the AI community. It can lead individuals to question their abilities and whether they truly belong in the AI field, even if they have achieved success.\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 078fd2d1-a75f-46ea-80a2-a304afd202a1.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: cafdfa80-e3ec-4877-825a-974ab062bd7c.\n",
      "> Parent node text: PAGE 37Overcoming Imposter \n",
      "SyndromeCHAPTER 11\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: cb9e385b-2d74-46b9-8452-3d41cd2609ef.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 33a81b18-db5a-4170-b2dd-9746b79325d7.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 977dfbae-8938-4e93-a6cc-ea55a9a46cce.\n",
      "> Parent node text: PAGE 37Overcoming Imposter \n",
      "SyndromeCHAPTER 11\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 07b83924-cf67-4952-bf27-b07f1b10e439.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "Who are some accomplished individuals who have experienced imposter syndrome?\n",
      "Sheryl Sandberg, Michelle Obama, Tom Hanks, and Mike Cannon-Brookes are some accomplished individuals who have experienced imposter syndrome.\n",
      "What is the first step to becoming good at AI?\n",
      "The first step to becoming good at AI is to suck at it.\n",
      "What are some common challenges in AI?\n",
      "Some common challenges in AI include the highly iterative nature of AI projects, difficulties in project management due to uncertainty in time estimates for achieving target accuracy, and technical challenges that many individuals face when working on AI projects.\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 078fd2d1-a75f-46ea-80a2-a304afd202a1.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 33a81b18-db5a-4170-b2dd-9746b79325d7.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "Is it normal to find parts of AI challenging?\n",
      "Yes, it is normal to find parts of AI challenging, as even accomplished individuals in the field have faced difficulties and struggles with various technical aspects of AI.\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 608bd2ca-f80e-4423-a79a-23b71fd1f85c.\n",
      "> Parent node text: PAGE 31Finding the Right \n",
      "AI Job for YouCHAPTER 9\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: e7fa6d16-64e1-47ac-9e52-3f0afd9cd16b.\n",
      "> Parent node text: If you’re leaving \n",
      "a job, exit gracefully. Give your employer ample notice, give your full effort...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: f6d586aa-09cb-403c-b76c-1d29984398fb.\n",
      "> Parent node text: PAGE 31Finding the Right \n",
      "AI Job for YouCHAPTER 9\n",
      "JOBS\n",
      "\n",
      "What is the right AI job for me?\n",
      "The right AI job for you is one that aligns with your skills, interests, and career goals. It should be a position where you can continue to grow, learn, and contribute effectively to the field of artificial intelligence.\n"
     ]
    }
   ],
   "source": [
    "for question in eval_questions_en:\n",
    "    with tru_recorder_automerging_en as recording:\n",
    "        response = automerging_query_engine_en.query(question)\n",
    "        print(question)\n",
    "        print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5404dec1-60ca-42fa-ac13-793a5423aa64",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>app_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Automerging Query Engine</th>\n",
       "      <td>0.8000</td>\n",
       "      <td>0.908333</td>\n",
       "      <td>0.916667</td>\n",
       "      <td>3.666667</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Automerging Query Engine_en</th>\n",
       "      <td>0.0375</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.840000</td>\n",
       "      <td>3.545455</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Context Relevance  Groundedness  \\\n",
       "app_id                                                         \n",
       "Automerging Query Engine                0.8000      0.908333   \n",
       "Automerging Query Engine_en             0.0375      0.333333   \n",
       "\n",
       "                             Answer Relevance   latency  total_cost  \n",
       "app_id                                                               \n",
       "Automerging Query Engine             0.916667  3.666667         0.0  \n",
       "Automerging Query Engine_en          0.840000  3.545455         0.0  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tru.get_leaderboard(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f545d41-0d98-446f-8214-8b59bef08d6c",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting dashboard ...\n",
      "Config file already exists. Skipping writing process.\n",
      "Credentials file already exists. Skipping writing process.\n",
      "Dashboard already running at path:   Network URL: http://198.18.0.1:8501\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Popen: returncode: None args: ['streamlit', 'run', '--server.headless=True'...>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# launches on http://localhost:8501/\n",
    "tru.run_dashboard()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "967d0a0d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
